Video display systems

ABSTRACT

A method for transmission of a sequence of high quality images for display on a visual display unit is described. It has particular application in the field of telepathology, where magnified images obtained by scanning a medical specimen on the stage of an optical microscope are transmitted from a local pathologist to a remote consultant for diagnosis. The method consists of the steps of advancing the field of view of the camera in discrete steps across the object being viewed to capture a sequence of contiguous images, temporarily storing a digital representation of a first of the images in a first buffer store and a digital representation of at least part of the next succeeding image in a second buffer store, controlling the data in the two buffers stores such that the data follows the leading edge of the advancing viewpoint and data in the first buffer store representing a trailing incremental strip of the first stored image is progressively discarded while data in the second buffer store representing a trailing incremental strip of the next contiguous image is transferred to a location in the first buffer store occupied by the data representing a leading incremental strip of the first stored image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to the transmission of a sequence of high qualityimages for display on a visual display unit. It has particularapplication in the field of telepathology where magnified imagesobtained by scanning a medical specimen on the stage of an opticalmicroscope are transmitted from a local pathologist to a remoteconsultant for diagnosis.

2. Discussion of the Background

For general diagnostic practice in tumour histopathology andcytopathology, because of the potentially serious consequences ofmisdiagnosis, it is hardly ever acceptable to examine just one or a fewstatic images from the microscope, no matter how high their individualquality may be. Instead, it is accepted that the diagnositician must befree to examine any part of the specimen, at any of the magnificationfactors which the microscope allows. Thus the remote consultant shouldhave the ability to ‘scan’ the field of view across the specimen on themicroscope stage; both along the left-right (-x) axis and the top-bottom(-y) axis of the field of view. The consultant should also be able tomove the stage in the longitudinal (-z) axis of the microscope to adjustthe focus.

Controlling the microscope stage through verbal instructions to a localpathologist is unacceptably slow and unreliable. Transmitting a completeset of images which together cover the whole specimen could be doneautomatically using a motorised stage and suitable camera controlsoftware, but would require the transmission of around 4000 images for ahistopathology section of 15 mm by 10 mm. This again is unacceptable.

Accordingly, remote control of the sender's microscope is virtuallyessential for a practical system.

One such system is described, for example, in U.S. Pat. Nos. 5,216,596and 5,297,034. In this known system, the magnified image of the specimenis recorded by a video camera and converted to an electronic videosignal which is then transmitted over a communication link to a remotevideo display monitor. Control signals are generated by a computerprocessing unit at the remote workstation for remotely controlling thefunctions of the microscope, including motorised stage movement,magnification, focus and illumination control.

The main problem with this known system is that the quality of the imageviewed by the remote consultant on the display monitor is well below thequality that would be seen by viewing the specimen directly through themicroscope.

There is an emerging consensus that 1024>768 is the minimum acceptablepixel format for display of diagnostic-quality images on ahigh-resolution colour monitor. This rules out the use of all analogvideo cameras, monitors and image compression/decompression devices(codecs) which are based on broadcast standards such as PAL or NTSC. Thebandwidth limitation imposed by the broadcase standards reduces theeffective pixel number in each image to about a quarter of the numberrequired for diagnostic resolution; in addition, in composite TVequipment, the colour resolution of the signal is further reduced by thechrominance subsampling.

Although digital videocameras with CCD chips are now available which aresuitable for capturing high-resolution microscope images, these cannotbe used to directly display images on composite video monitors. Insteadthey are designed to work with image digitisers (frame grabbers) bymeans of which a digital representation of the image is stored in RAM,or in storage medium such as magnetic disc or CD ROM. To visuallydisplay such an image, it must be written to an ‘RGB’ colour monitorwith a display driver capable of handling images of at least 1024×768pixels at 8 or more bits/colour channel.

The overriding problem in the use of digital cameras in a telepathologydiagnostic system is that, unlike analog video cameras, digital camerascannot provide images at video rate: in fact a maximum rgb frame rate ofabout {fraction (1/25)} of video rate (2/sec), is typical, and inseveral cases the frame rate is less than 1/sec.

Although standard analog TV videocodec technology can be used tocompress the video images such that the required bandwidth is reduced toa practical level (say 384 kbit/sec), if 50 images must be transmittedper second, this allows 384/50 or about 8 kbits per image. Since thefinal image is normally reduced to a size of (512×384) 24 bit pixels (orabout 4 Mbit), this requires the codec to perform (lossy) imagecompression of 500:1. This can only be achieved at the expense ofsignificant image degradation.

To achieve smooth scrolling of the field of view of a digitalvideocamera across a specimen, the image refresh rate in the viewportmust be comparable to the flicker fusion frequency of normal humanvision, say 30/sec. But because the maximum image capture rate of thedigital camera is much less than this (of the order of 1 frame/sec), itis not possible to simply grab, transmit, and display a stream ofcomplete high-resolution images as the specimen is scanned under theobjective. To overcome this fundamental difficulty without loss of imageresolution, the present invention makes use of the fact that duringscrolling, between one screen refresh and the next, although the rgbvalues of every pixel will in general be changed, the bulk of thedisplay if simply shifted slightly in a vertical or horizontal directionso that the information content of the image as a whole remains almostconstant.

SUMMARY OF THE INVENTION

According to the present invention there is provided a method ofdisplaying a sequence of images from a digital videocamera on a videodisplay unit, the method comprising advancing the field of view of thecamera in discrete steps across the object being viewed to capture asequence of contiguous images, temporarily storing a digitalrepresentation of a first of the images in a first buffer store and adigital representation of at least part of the next succeeding image ina second buffer store, controlling the data in the two buffer storessuch that the data follows the leading edge of the advancing viewportand data in the first store representing a trailing incremental strip ofthe first stored image is progressively discarded while data in thesecond store representing a trailing incremental strip of the nextcontiguous image is transferred to a location if the first storeoccupied by the data representing a leading incremental strip of thefirst stored image, and reading out the contents of the first bufferstore to the display unit.

The second buffer may either hold data representing a complete new imagein the camera viewport, or data representing an increment of ‘scrollquanta’ of the image. Where the second buffer holds an entire image, anintermediate or ‘scroll’ buffer containing the ‘scroll quanta’ may beinserted between the first and second buffers. Where the second storecontains an entire image, the incrementing is continued until the entirecontents of the first buffer store representing the original imagestored therein have been discarded, and a new image is then grabbed intothe second buffer store and the process is repeated. Alternatively, thenew image is stored in a third buffer store and the contents of thesecond buffer store are then replenished continuously with incrementsfrom the third buffer store. In either case, the specimen appears to bescrolling smoothly across the field of view while in reality it is movedin a series of quick steps followed by capture of contiguous images.

The data entered into the first buffer representing the trailingincremental strip of the image stored in the second buffer store can beentered directly from the second buffer store or through an intermediatescroll buffer containing only the incremental portion of the image beingtransferred.

When scrolling in the y-direction, the progressive replenishment of theimage in the first store is performed line by line or in multiples ofone complete line, whereas during x-scrolling each line of the displayedimage is created as a composite of the corresponding lines in the imagebuffers.

For a colour rgb image, first and second buffers can be provided foreach of the three colour components, and the readout is to a singlecommon display buffer.

BRIEF DESCRIPTION OF THE DRAWINGS

By way of example only, an embodiment of the invention will now bedescribed with reference to the accompanying drawings in which:

FIG. 1 is a schematic block diagram of a remote control telepathologysystem; and

FIG. 2 is a diagram illustrating the y-scrolling of the image in thesystem of FIG. 1.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to these figures, the pathological specimen to be viewed isplaced on a microscope stage 10 beneath an objective lens 11 of themicroscope 17 and illuminated by a light source 16. A digital videocamer 12 with a CCD chip suitable for capturing high-resolutionmicroscope images is fitted to the microscope.

A digital interface card (frame grabber) 13 retrieves and stores adigital representation of the image viewed by the camera 12, and thisdigital representation is processed by a local computer 14 andtransmitted to a remote computer 15 operated by a consultant.

The remote computer 15 is linked through a display driver 18 to a videodisplay unit 19.

The microscope stage 10 is remotely controlled by the consultant througha stepper motor unit 20. A similar situation holds in the case where theconsultant wishes to use the computer 15 to control a local microscope(not shown). As the consultant drives the microscope stage in discretesteps under the (fixed) objective 11 of the microscope 17, it appears asthough a viewport of fixed size is being moved (scrolled) smoothlyacross a single very large image of the specimen (referred to herein asa ‘virtual digital image’).

The manner in which apparent scrolling of the viewport across thevirtual digital image is achieved is illustrated in FIG. 2. Scrollingcan take place in both directions along trajectories parallel to eitherthe ‘y’-axis or to the ‘x’-axis. For simplicity we will first describethe method used for downward scrolling across a ‘virtual digital image’made of uncompressed monochrome or interleaved-colour images. Thismethod can be easily extended to scrolling in any direction across anytype of digital image, including compressed and non-interleaved colourimages, and including images with any number of bits per pixel.

In the following description, the number of bits/pixel in eachuncompressed image is called ‘nbits’. The number of horizontal ‘lines’in each image will be denoted by ‘ymax’, and the number of pixels withineach of these horizontal ‘lines’ by ‘xmax’. Thus xmax, ymax are thedigital width and height of the images respectively and the buffer sizerequired to hold a single image is given by the product (xmax, ymax,nbits).

FIG. 2 schematically represents two contiguous digital images,continuous in the specimen's ‘y’-direction which have been acquired intobuffer stores buf_(—)0 and buf_(—)1. The data buffers are continuousblocks of RAM, allocated at start-up by the operating system of thecomputer at the request of the controlling program. The individual (x)scan-lines in the images are denoted a, b, c . . . and A, B, C . . . inbuf_(—)0 and buf_(—)1 respectively. Scanning commences at a point intime when the image in buf_(—)0 is already being displayed.

Scrolling on the Local Workstation

Each elemental downward ‘scroll event’ can be achieved through thefollowing series of steps:

(1) left-shift the contents of buf_(—)0 by N(nbits.xmax) bits, where Nrepresents the number of lines in each scroll event. (Note that N shouldbe chosen such that ymax is an integer multiple of N.)

(2) copy the first N(nbits.xmax) from buf_(—)1 into a temporary ‘scrollquantum’ buffer (not shown).

(3) write the contents of the scroll quantum buffer into buf_(—)0,starting at an offset in buf_(—)0 given by (xmax(ymax-N)nbits.

(3a) write the contents of the scroll quantum buffer to the output commsport.

(4) left-shift the contents of buf_(—)1 by N(nbits.xmax)

(5) display the contents of buf_(—)0

If it is not necessary to transmit the scrolling viewport to anotherworkstation over a LAN or WAN connection, then step 3a will be omitted.In this case the scroll quantum buffer can also be dispenses with sincethe scroll quantum can be copied directly from buf ₁₃ 1 to buf_(—)0.

Steps 1-5 can be repeated to continue scrolling until (ymax/N) scrollevents have been competed. At this point there is no more useful data inbuf_(—)1 and the scroll process must pause while a new image is acquiredinto buf_(—)1. Scrolling can then resume as before until a further(ymax/N) scroll events have occurred, etc. If it is desired to reducethe slight irregularity which accompanies this type of scrolling. It isa simple matter to use three buffers instead of two, at the cost of morememory allocation and slightly greater complexity in programming.

In practice, by writing directly to the display hardware, this processcan be accomplished within 15 msec, easily meeting the need to refreshthe display at 30 frames/sec to create the impression of smoothscrolling. To scroll across a full viewport dimension in this way takesapproximately 12 sec. using compiled Visual C++ code, a standard rgbdisplay board and an Intel P166 cpu. If two complete new lines (a, b, cd, . . . ) are copied to buf_(—)0 in each ‘scroll event’, then the timetaken to scan one complete frame height is halved without noticeableloss of smoothness in scrolling. Further incrementing the ‘line quantum’gives a slight but noticeable jerkiness to the scrolling, which may beacceptable as the price to pay for faster scrolling. For a colour image,six image buffers would be required to accommodate the colour imageplanes received from the camera, the display buffer buf_(—)2 holding thecomposite image to be displayed on the monitor at any instant.

Scrolling on the Remote Workstation

Examination of the five steps which make up a single local scroll eventshows that a similar sequence can be carried out on a remote host, buthere instead of holding an entire image, buf_(—)1 is replaced by theinput buffer provided by the communications software layer which holdsthe ‘scroll quanta’ of size N(xmax.nbits) bits. Starting from the samepoint as before, with both computers displaying the same initial imagefrom their respective buf_(—)0 memories, the sequence on the remote hostfor a parallel downward scroll event then becomes

(1) left-shift contents of the remote buf_(—)0 by N(nbits.xmax) bits,where N represents the number of lines in each scroll event.

(2) read a ‘scroll quantum’ of N(nbits.xmax) bits from the comms inputdata stream into a remote buf_(—)0, starting at an offset in the remotebuf_(—0) given by (xmax(ymax-N)nbits.

(3) display the contents of buf_(—)0.

In this way the communications link is used to transmit a stream ofsuccessive ‘scroll quanta’. While not essential to the concept ofviewport scrolling presented here, this method has two advantagescompared with the alternative strategy of transmitting a succession ofentire images to the remote host:

a) following initiation of the scrolling process via the local host'sgui, scrolling begins earlier on the remote host. This is because thereis no need to wait for an entire image to be transmitted beforebeginning scrolling on the remote host.

b) because the image is transmitted in small ‘chunks’ rather than as onelong bitstream, buffer management and error handling are made easier onthe remote host.

Movement of the specimen under the objective lens 11 in all threeorthogonal axes is brought about through activation of one or more ofthree stepper-motors provided with the motorised stage. The threestepper-motors are driven directly from the proprietary stage controllerhardware which in turn receives appropriate control signals in the formof ASCII strings via an RS232 port of the local computer 14.

The microscope stage 10 is controlled by the consultant using a mouse toactivate one of the four arrow symbols which the gui provides (‘scrollforward, scroll back, scroll left, scroll right’). The key feature ofthis type of control is that the microscope does not move smoothly atconstant speed: on the contrary, the stage is made to move in stepscorresponding to the absolute height (y₀) or width (x₀) of the region inthe viewport frame as appropriate to the selected direction of travel.For example, suppose that the camera/framegrabber in use provides animage (viewport) of height 768 pixels, and that calibration of thesystem with stage micrometer while using a specific objective lens showsthat 1 mm in the focal plane corresponding to 218 pixels. It followsthat y₀=3.52 mm, and this will be programmed into the startup scriptduring the initial calibration. If now the mouse is used to select the‘scroll forward’ arrow, the stage will be made to move forward in stepsof 3.52 mm. Immediately that each step is completed (approx 1 sec), animage is grabbed and compressed to RAM (0.5 sec). Once an initial staticimage has been transferred to the remote display buffer, all that isrequired is that this buffer is treated in the same way as buf_(—)0 inFIG. 2, ie it must be updated by incorporation of the ‘scroll quanta’which it gets from the local computer.

When one of the ‘arrow keys’ is activated, then the stage immediatelymoves by one viewport height (or width) in the appropriate direction andan image is grabbed to buf_(—)1. No further movement of the stage occursuntil the viewport is scrolled completely out o fbuf_(—)0, whereupon thestage is made to jump ahead again.

When scrolling in the x-direction, it is no longer possible to simplyshift the entire contents of the store to the right or left whenincrementing the stored data. Instead, the data representing anincremental strip in the x-direction must be retrieved from each of theymax lines in the appropriate store and either discarded or combinedwith the data of the corresponding ymax lines in the other store to forman up-dated image.

Other applications of the described system will exist wherever readoutof data from a videocamera is slow compared to the movement of a fixedscene across the field of view. In the present example, the image of thespecimen is moved across the CCD camera chip in a ‘salatory’ or‘jumping’ motion so that a smooth, continuous motion can be accuratelydisplayed on the VDU. The same principle might be used to move thecamera itself in this way across a static object to produce a similareffect. For example, if the camera were made to perform a series ofrotations through a fixed angular distance about a static axis, pausingon the way to take a series of static images of a panoramic scene, thenthese images could be processes in the way described above toreconstruct the visual effect of a smooth panning motion across thescene at almost any desired speed. A similar process could be used tomake smooth images in aerial photography, virtual reality constructionsetc., provided that the object in view is not changing appreciablyduring the period in which the images are acquired.

What is claimed is:
 1. Method of displaying a sequence of images from adigital videocamera on a video display unit, the method beingcharacterized by the steps of advancing the field of view of the camerais discrete steps across the object being viewed to capture a sequenceof contiguous images, temporarily storing a digital representation of afirst of the images in a first buffer store and a digital representationof at least part of the next succeeding image in a second buffer store,controlling the data in the two buffer stores such that the data followsthe leading edge of the advancing viewport and data in the first storerepresenting a trailing increment strip of the first stored image isprogressively discarded while data in the second store representing atrailing increment strip of the next contiguous image is transferred toa location in the first store occupied by the data representing aleading incremental strip of the first stored image, and reading out thecontents of the first buffer store to the display unit.
 2. Methodaccording to claim 1, characterized in that where the second bufferholds an entire image, an intermediate ‘scroll’ buffer containing the‘scroll quanta’ is inserted between the first and the second buffers. 3.Method according to claim 2, in which the image is a color RGB image,characterized in that the first and second buffers are provided for eachof the three color components, and that the readout is to a singlecommon display buffer.
 4. Use of a method according to claim 2 in asystem for transferring pathological information, such as scannedimages, from a local to a remote location.
 5. Method according to claim1, characterized in that the second buffer contains an entire image,that the incrementation is continued until the entire contents of thefirst buffer store representing the original image stored therein havebeen discarded, and that a new image is then grabbed into the secondbuffer store and the process is repeated.
 6. Method according to claim5, in which the image is a color RGB image, characterized in that thefirst and second buffers are provided for each of the three colorcomponents, and that the readout is to a single common display buffer.7. Use of a method according to claim 5 in a system for transferringpathological information, such as scanned images, from a local to aremote location.
 8. Method according to claim 1, characterized in thatthe new image is stored in a third image store and that the contents ofthe second buffer store are then replenished continuously withincrements from the third buffer store.
 9. Method according to claim 8in which the image is a color RGB image, characterized in that the firstand second buffers are provided for each of the three color components,and that the readout is to a single common display buffer.
 10. Use of amethod according to claim 8 in a system for transferring pathologicalinformation, such as scanned images, from a local to a remote location.11. Method according to claim 1, in which the image is a colour RGBimage, characterized in that the first and second buffers are providedfor each of the three colour components, and that the readout is to asingle common display buffer.
 12. Use of a method according to claim 11in a system for transferring pathological information, such as scannedimages, from a local to a remote location.
 13. Use of a method accordingto claim 1 in a system for transferring pathological information, suchas scanned images, from a local to a remote location.